Method and apparatus for in-camera night lapse video

ABSTRACT

A method and apparatus for night lapse video capture are disclosed. The apparatus includes an image sensor, a processor, a denoiser, and an encoder. The image sensor is configured to capture image data. The processor is configured to process the image data using an image processing pipeline. The denoiser is configured to perform denoising based on a motion estimation that is based on a comparison of a portion of the first image and a portion of the second image. The encoder is configured to encode the denoised image in a video format. The encoder is configured to store a video file in the video format.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.17/380,392, filed Jul. 20, 2021, which is a continuation of U.S. patentapplication Ser. No. 16/391,869, filed Apr. 23, 2019, now U.S. Pat. No.11,070,765, the entire disclosures of which are incorporated byreference herein.

TECHNICAL FIELD

This disclosure relates to digital image and video processing, includingin-camera night lapse video capture.

BACKGROUND

Image capture devices, such as cameras, may capture content as images,video, or both. Typical cameras may have a night photo mode thatgenerates photos with long shutter times, long integration times, orboth, that require substantial offline processing in order to achieve anight lapse effect. Typical image capture devices do not support videomodes having a long integration time, and therefore it is not possibleto capture a night lapse video in-camera in real-time.

SUMMARY

Disclosed herein are implementations of in-camera night lapse videocapture. An aspect of the implementations disclosed herein may includean image capture device. The image capture device may have a nightexposure and a non-night exposure. A duration of the night exposure maybe greater than a duration of the non-night exposure. The image capturedevice may include an image sensor, an image processor, and a videoencoder. The image sensor may be configured to capture image data atnight. The image data may include a first image that is temporallyprecedent to a second image. The image processor may be configured todetermine a motion estimation. The motion estimation may be based on acomparison of a portion of the first image and a portion of the secondimage. The portion of the first image may correspond to the portion ofthe second image. The image processor may be configured to obtain a maskbased on the motion estimation. The image processor may be configured tosubtract the mask from the second image to obtain a denoised image. Thevideo encoder may be configured to receive the denoised image from theimage processor. The video encoder may be configured to encode thedenoised image in a video format. The video encoder may be configured tooutput a video file in the video format.

An aspect may include an in-camera method for capturing night lapsevideo. The method may include capturing an image and a reference image.The reference image may be temporally precedent to the image. The methodmay include processing the image and the reference image using an imageprocessing pipeline. The method may include denoising the image. Themethod may include encoding the denoised image. The denoised image maybe encoded in a video format. The method may include outputting a videofile in the video format.

An aspect may include an image capture device that includes an imagesensor, an image processor, and a video encoder. The image sensor may beconfigured to capture image data. The image data may include a firstimage that is temporally precedent to a second image. The imageprocessor may be configured to determine a motion estimation. The motionestimation may be based on a comparison of a portion of the first imageand a portion of the second image. The portion of the first image maycorrespond to the portion of the second image. The image processor maybe configured to subtract a mask from the second image to obtain adenoised image. The mask may be based on the motion estimation. Thevideo encoder may be configured to receive the denoised image from theimage processor. The video encoder may be configured to encode thedenoised image in a video format. The video encoder may be configured tooutput a video file in the video format.

An aspect may include a method that includes obtaining an image and areference image. The reference image may be temporally precedent to theimage. The method may include processing the image and the referenceimage using an image processing pipeline. The image may be a 15-bitBayer image. Processing the image may include converting the 15-bitBayer image to a 14-bit Bayer image. Processing the image may includeconverting the 14-bit Bayer image to a 14-bit RGB image. Processing theimage may include converting the 14-bit Bayer image to a 10-bit YUVimage. The method may include denoising the 10-bit YUV image to obtain adenoised image. The method may include encoding the denoised image in avideo format. The method may include outputting a video file in thevideo format.

An aspect may include an image capture device that includes an imagesensor, a processor, a denoiser, and an encoder. The image sensor may beconfigured to obtain an image and a reference image. The reference imagemay be temporally precedent to the image. The processor may beconfigured to process the image and the reference image using an imageprocessing pipeline. The processor may be configured to convert a 15-bitBayer reference image to a 14-bit Bayer reference image. The processormay be configured to convert the 14-bit Bayer reference image to a14-bit RGB reference image. The processor may be configured to convertthe 14-bit RGB reference image to a 10-bit YUV reference image. Thedenoiser may be configured to denoise the image based on a motionestimation. The motion estimation may be based on a comparison of aportion of the image and a portion of the reference image. The portionof the image may correspond to the portion of the reference image. Theencoder may be configured to encode the denoised image in a videoformat. The encoder may be configured to output a video file in thevideo format.

An aspect may include an image capture device that includes an imagesensor, a processor, a denoiser, and an encoder. The image sensor may beconfigured to obtain an image and a reference image. The reference imagemay be temporally precedent to the image. The processor may beconfigured to process the image and the reference image using an imageprocessing pipeline. The processor may be configured to obtain a 10-bitYUV image. The processor may be configured to obtain a 10-bit YUV image.The denoiser may be configured to denoise the 10-bit YUV image to obtaina denoised image. The 10-bit YUV image may be denoised based on a motionestimation that is based on a comparison of a portion of the 10-bit YUVimage and a portion of the 10-bit YUV reference image. The portion ofthe 10-bit YUV image may correspond to the portion of the 10-bit YUVreference image. The encoder may be configured to encode the denoisedimage in a video format. The encoder may be configured to output a videofile in the video format.

An aspect may include a method that includes obtaining an image and areference image. The method may include processing the image and thereference image using an image processing pipeline. The image may be aBayer image. Processing the image may include converting the Bayer imageto a YUV image. The method may include denoising the YUV image to obtaina denoised image. The method may include encoding the denoised image ina video format. The method may include storing a video file in the videoformat.

An aspect may include an image capture device that includes an imagesensor, a processor, a denoiser, and an encoder. The image sensor may beconfigured to obtain an image and a reference image. The processor maybe configured to process the image and the reference image using animage processing pipeline. The processor may be configured to convert aBayer reference image to a YUV reference image. The denoiser may beconfigured to denoise the image based on a motion estimation. The motionestimation may be based on a comparison of a portion of the image and aportion of the reference image. The encoder may be configured to encodethe denoised image in a video format. The encoder may be configured tostore a video file in the video format.

An aspect may include an image capture device that includes an imagesensor, a processor, a denoiser, and an encoder. The image sensor may beconfigured to obtain an image and a reference image. The processor maybe configured to process the image and the reference image using animage processing pipeline. The processor may be configured to obtain aYUV image. The processor may be configured to obtain a YUV image. Thedenoiser may be configured to denoise the YUV image to obtain a denoisedimage. The encoder may be configured to encode the denoised image in avideo format. The encoder may be configured to store a video file in thevideo format.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is best understood from the following detaileddescription when read in conjunction with the accompanying drawings. Itis emphasized that, according to common practice, the various featuresof the drawings are not to-scale. On the contrary, the dimensions of thevarious features are arbitrarily expanded or reduced for clarity.

FIGS. 1A-D are isometric views of an example of an image capture device.

FIGS. 2A-B are isometric views of another example of an image capturedevice.

FIG. 2C is a cross-sectional view of the image capture device of FIGS.2A-B.

FIGS. 3A-B are block diagrams of examples of image capture systems.

FIG. 4 is a functional block diagram of an example of an in-camera imageprocessing and coding pipeline portion for night lapse video.

FIG. 5 is a functional block diagram of another example of an in-cameraimage processing and coding pipeline portion for night lapse video.

FIG. 6 is a functional block diagram of another example of an in-cameraimage processing and coding pipeline portion for night lapse video.

FIG. 7 is a functional block diagram of an example of an in-camera videoprocessing and coding pipeline with real-time video preview.

FIG. 8 is a flow diagram of an in-camera video processing method fornight lapse video.

DETAILED DESCRIPTION

Implementations described herein may include an in-camera video modethat applies an image processing pipeline configured for a photo mode inconjunction with a temporal noise reduction unit and a video encodingunit. Image quality may be retained by the application of the photo modepipeline. In the implementations described herein, the image sensors maybe configured to support long integration times of 30 seconds or morefor in-camera night lapse video capture. In automatic mode, the shuttertime, the integration time, or both, may be set up to 10 seconds. Inmanual mode, the shutter time, the integration time, or both, may be setto 2 seconds, 5 seconds, 10 seconds, 15 seconds, 20 seconds, or 30seconds. In one or more implementations described herein, the videoencoder may determine a group of frames (GOP) as an intra-coded image(I-Frame) only, where an I-Frame is an image that is coded independentlyof all other images. In one or more implementations described herein,the video encoder may determine a GOP as an I-Frame and apredictive-coded image (P-Frame) as a function of a sceneclassification, where a P-Frame may include motion compensationinformation relative to previously decoded images. The video encoder maybe configured to set a playback speed as a function of ISO, integrationtime, object motion, or any combination thereof. An example of objectmotion may be the moon moving across the sky, light painting, or drivingthrough a city at night.

FIGS. 1A-D are isometric views of an example of an image capture device100. The image capture device 100 may include a body 102 having a lens104 structured on a front surface of the body 102, various indicators onthe front of the surface of the body 102 (such as LEDs, displays, andthe like), various input mechanisms (such as buttons, switches, andtouch-screen mechanisms), and electronics (e.g., imaging electronics,power electronics, etc.) internal to the body 102 for capturing imagesvia the lens 104 and/or performing other functions. The image capturedevice 100 may be configured to capture images and video and to storecaptured images and video for subsequent display or playback.

The image capture device 100 may include various indicators, includingLED lights 106 and LCD display 108. The image capture device 100 mayalso include buttons 110 configured to allow a user of the image capturedevice 100 to interact with the image capture device 100, to turn theimage capture device 100 on, to operate latches or hinges associatedwith doors of the image capture device 100, and/or to otherwiseconfigure the operating mode of the image capture device 100. The imagecapture device 100 may also include a microphone 112 configured toreceive and record audio signals in conjunction with recording video.

The image capture device 100 may include an I/O interface 114 (e.g.,hidden as indicated using dotted lines). As best shown in FIG. 1B, theI/O interface 114 can be covered and sealed by a removable door 115 ofthe image capture device 100. The removable door 115 can be secured, forexample, using a latch mechanism 115 a (e.g., hidden as indicated usingdotted lines) that is opened by engaging the associated button 110 asshown.

The removable door 115 can also be secured to the image capture device100 using a hinge mechanism 115 b, allowing the removable door 115 topivot between an open position allowing access to the I/O interface 114and a closed position blocking access to the I/O interface 114. Theremovable door 115 can also have a removed position (not shown) wherethe entire removable door 115 is separated from the image capture device100, that is, where both the latch mechanism 115 a and the hingemechanism 115 b allow the removable door 115 to be removed from theimage capture device 100.

The image capture device 100 may also include another microphone 116integrated into the body 102 or housing. The front surface of the imagecapture device 100 may include two drainage ports as part of a drainagechannel 118. The image capture device 100 may include an interactivedisplay 120 that allows for interaction with the image capture device100 while simultaneously displaying information on a surface of theimage capture device 100. As illustrated, the image capture device 100may include the lens 104 that is configured to receive light incidentupon the lens 104 and to direct received light onto an image sensorinternal to the lens 104.

The image capture device 100 of FIGS. 1A-D includes an exterior thatencompasses and protects internal electronics. In the present example,the exterior includes six surfaces (i.e. a front face, a left face, aright face, a back face, a top face, and a bottom face) that form arectangular cuboid. Furthermore, both the front and rear surfaces of theimage capture device 100 are rectangular. In other embodiments, theexterior may have a different shape. The image capture device 100 may bemade of a rigid material such as plastic, aluminum, steel, orfiberglass. The image capture device 100 may include features other thanthose described here. For example, the image capture device 100 mayinclude additional buttons or different interface features, such asinterchangeable lenses, cold shoes and hot shoes that can add functionalfeatures to the image capture device 100, etc.

The image capture device 100 may include various types of image sensors,such as a charge-coupled device (CCD) sensors, active pixel sensors(APS), complementary metal-oxide-semiconductor (CMOS) sensors, N-typemetal-oxide-semiconductor (NMOS) sensors, and/or any other image sensoror combination of image sensors.

Although not illustrated, in various embodiments, the image capturedevice 100 may include other additional electrical components (e.g., animage processor, camera SoC (system-on-chip), etc.), which may beincluded on one or more circuit boards within the body 102 of the imagecapture device 100.

The image capture device 100 may interface with or communicate with anexternal device, such as an external user interface device, via a wiredor wireless computing communication link (e.g., the I/O interface 114).The user interface device may, for example, be the personal computingdevice 360 described below with respect to FIG. 3B. Any number ofcomputing communication links may be used. The computing communicationlink may be a direct computing communication link or an indirectcomputing communication link, such as a link including another device ora network, such as the internet, may be used.

In some implementations, the computing communication link may be a Wi-Filink, an infrared link, a Bluetooth (BT) link, a cellular link, a ZigBeelink, a near field communications (NFC) link, such as an ISO/IEC 20643protocol link, an Advanced Network Technology interoperability (ANT+)link, and/or any other wireless communications link or combination oflinks.

In some implementations, the computing communication link may be an HDMIlink, a USB link, a digital video interface link, a display portinterface link, such as a Video Electronics Standards Association (VESA)digital display interface link, an Ethernet link, a Thunderbolt link,and/or other wired computing communication link.

The image capture device 100 may transmit images, such as panoramicimages, or portions thereof, to the user interface device (not shown)via the computing communication link, and the user interface device maystore, process, display, or a combination thereof the panoramic images.

The user interface device may be a computing device, such as asmartphone, a tablet computer, a phablet, a smart watch, a portablecomputer, and/or another device or combination of devices configured toreceive user input, communicate information with the image capturedevice 100 via the computing communication link, or receive user inputand communicate information with the image capture device 100 via thecomputing communication link.

The user interface device may display, or otherwise present, content,such as images or video, acquired by the image capture device 100. Forexample, a display of the user interface device may be a viewport intothe three-dimensional space represented by the panoramic images or videocaptured or created by the image capture device 100.

The user interface device may communicate information, such as metadata,to the image capture device 100. For example, the user interface devicemay send orientation information of the user interface device withrespect to a defined coordinate system to the image capture device 100,such that the image capture device 100 may determine an orientation ofthe user interface device relative to the image capture device 100.

Based on the determined orientation, the image capture device 100 mayidentify a portion of the panoramic images or video captured by theimage capture device 100 for the image capture device 100 to send to theuser interface device for presentation as the viewport. In someimplementations, based on the determined orientation, the image capturedevice 100 may determine the location of the user interface deviceand/or the dimensions for viewing of a portion of the panoramic imagesor video.

The user interface device may implement or execute one or moreapplications to manage or control the image capture device 100. Forexample, the user interface device may include an application forcontrolling camera configuration, video acquisition, video display, orany other configurable or controllable aspect of the image capturedevice 100.

The user interface device, such as via an application, may generate andshare, such as via a cloud-based or social media service, one or moreimages, or short video clips, such as in response to user input. In someimplementations, the user interface device, such as via an application,may remotely control the image capture device 100 such as in response touser input.

The user interface device, such as via an application, may displayunprocessed or minimally processed images or video captured by the imagecapture device 100 contemporaneously with capturing the images or videoby the image capture device 100, such as for shot framing, which may bereferred to herein as a live preview, and which may be performed inresponse to user input. In some implementations, the user interfacedevice, such as via an application, may mark one or more key momentscontemporaneously with capturing the images or video by the imagecapture device 100, such as with a tag, such as in response to userinput.

The user interface device, such as via an application, may display, orotherwise present, marks or tags associated with images or video, suchas in response to user input. For example, marks may be presented in acamera roll application for location review and/or playback of videohighlights.

The user interface device, such as via an application, may wirelesslycontrol camera software, hardware, or both. For example, the userinterface device may include a web-based graphical interface accessibleby a user for selecting a live or previously recorded video stream fromthe image capture device 100 for display on the user interface device.

The user interface device may receive information indicating a usersetting, such as an image resolution setting (e.g., 3840 pixels by 2160pixels), a frame rate setting (e.g., 60 frames per second (fps)), alocation setting, and/or a context setting, which may indicate anactivity, such as mountain biking, in response to user input, and maycommunicate the settings, or related information, to the image capturedevice 100.

FIGS. 2A-B illustrate another example of an image capture device 200.The image capture device 200 includes a body 202 and two camera lenses204, 206 disposed on opposing surfaces of the body 202, for example, ina back-to-back or Janus configuration.

The image capture device may include electronics (e.g., imagingelectronics, power electronics, etc.) internal to the body 202 forcapturing images via the lenses 204, 206 and/or performing otherfunctions. The image capture device may include various indicators suchas an LED light 212 and an LCD display 214.

The image capture device 200 may include various input mechanisms suchas buttons, switches, and touchscreen mechanisms. For example, the imagecapture device 200 may include buttons 216 configured to allow a user ofthe image capture device 200 to interact with the image capture device200, to turn the image capture device 200 on, and to otherwise configurethe operating mode of the image capture device 200. In animplementation, the image capture device 200 includes a shutter buttonand a mode button. It should be appreciated, however, that, in alternateembodiments, the image capture device 200 may include additional buttonsto support and/or control additional functionality.

The image capture device 200 may also include one or more microphones218 configured to receive and record audio signals (e.g., voice or otheraudio commands) in conjunction with recording video.

The image capture device 200 may include an I/O interface 220 and aninteractive display 222 that allows for interaction with the imagecapture device 200 while simultaneously displaying information on asurface of the image capture device 200.

The image capture device 200 may be made of a rigid material such asplastic, aluminum, steel, or fiberglass. In some embodiments, the imagecapture device 200 described herein includes features other than thosedescribed. For example, instead of the I/O interface 220 and theinteractive display 222, the image capture device 200 may includeadditional interfaces or different interface features. For example, theimage capture device 200 may include additional buttons or differentinterface features, such as interchangeable lenses, cold shoes and hotshoes that can add functional features to the image capture device 200,etc.

FIG. 2C is a cross-sectional view of the image capture device 200 ofFIGS. 2A-B. The image capture device 200 is configured to capturespherical images, and accordingly, includes a first image capture device224 and a second image capture device 226. The first image capturedevice 224 defines a first field-of-view 228 as shown in FIG. 2C andincludes the lens 204 that receives and directs light onto a first imagesensor 230.

Similarly, the second image capture device 226 defines a secondfield-of-view 232 as shown in FIG. 2C and includes the lens 206 thatreceives and directs light onto a second image sensor 234. To facilitatethe capture of spherical images, the image capture devices 224, 226 (andrelated components) may be arranged in a back-to-back (Janus)configuration such that the lenses 204, 206 face in generally oppositedirections.

The fields-of-view 228, 232 of the lenses 204, 206 are shown above andbelow boundaries 236, 238, respectively. Behind the first lens 204, thefirst image sensor 230 may capture a first hyper-hemispherical imageplane from light entering the first lens 204, and behind the second lens206, the second image sensor 234 may capture a secondhyper-hemispherical image plane from light entering the second lens 206.

One or more areas, such as blind spots 240, 242 may be outside of thefields-of-view 228, 232 of the lenses 204, 206 so as to define a “deadzone.” In the dead zone, light may be obscured from the lenses 204, 206and the corresponding image sensors 230, 234, and content in the blindspots 240, 242 may be omitted from capture. In some implementations, theimage capture devices 224, 226 may be configured to minimize the blindspots 240, 242.

The fields-of-view 228, 232 may overlap. Stitch points 244, 246,proximal to the image capture device 200, at which the fields-of-view228, 232 overlap may be referred to herein as overlap points or stitchpoints. Content captured by the respective lenses 204, 206, distal tothe stitch points 244, 246, may overlap.

Images contemporaneously captured by the respective image sensors 230,234 may be combined to form a combined image. Combining the respectiveimages may include correlating the overlapping regions captured by therespective image sensors 230, 234, aligning the captured fields-of-view228, 232, and stitching the images together to form a cohesive combinedimage.

A slight change in the alignment, such as position and/or tilt, of thelenses 204, 206, the image sensors 230, 234, or both, may change therelative positions of their respective fields-of-view 228, 232 and thelocations of the stitch points 244, 246. A change in alignment mayaffect the size of the blind spots 240, 242, which may include changingthe size of the blind spots 240, 242 unequally.

Incomplete or inaccurate information indicating the alignment of theimage capture devices 224, 226, such as the locations of the stitchpoints 244, 246, may decrease the accuracy, efficiency, or both ofgenerating a combined image. In some implementations, the image capturedevice 200 may maintain information indicating the location andorientation of the lenses 204, 206 and the image sensors 230, 234 suchthat the fields-of-view 228, 232, stitch points 244, 246, or both may beaccurately determined, which may improve the accuracy, efficiency, orboth of generating a combined image.

The lenses 204, 206 may be laterally offset from each other, may beoff-center from a central axis of the image capture device 200, or maybe laterally offset and off-center from the central axis. As compared toimage capture devices with back-to-back lenses, such as lenses alignedalong the same axis, image capture devices including laterally offsetlenses may include substantially reduced thickness relative to thelengths of the lens barrels securing the lenses. For example, theoverall thickness of the image capture device 200 may be close to thelength of a single lens barrel as opposed to twice the length of asingle lens barrel as in a back-to-back configuration. Reducing thelateral distance between the lenses 204, 206 may improve the overlap inthe fields-of-view 228, 232.

Images or frames captured by the image capture devices 224, 226 may becombined, merged, or stitched together to produce a combined image, suchas a spherical or panoramic image, which may be an equirectangularplanar image. In some implementations, generating a combined image mayinclude three-dimensional, or spatiotemporal, noise reduction (3DNR). Insome implementations, pixels along the stitch boundary may be matchedaccurately to minimize boundary discontinuities.

FIGS. 3A-B are block diagrams of examples of image capture systems.

Referring first to FIG. 3A, an image capture system 300 is shown. Theimage capture system 300 includes an image capture device 310 (e.g., acamera or a drone), which may, for example, be the image capture device200 shown in FIGS. 2A-C.

The image capture device 310 includes a processing apparatus 312 that isconfigured to receive a first image from a first image sensor 314 andreceive a second image from a second image sensor 316. The image capturedevice 310 includes a communications interface 318 for transferringimages to other devices. The image capture device 310 includes a userinterface 320 to allow a user to control image capture functions and/orview images. The image capture device 310 includes a battery 322 forpowering the image capture device 310. The components of the imagecapture device 310 may communicate with each other via the bus 324.

The processing apparatus 312 may be configured to perform image signalprocessing (e.g., filtering, tone mapping, stitching, and/or encoding)to generate output images based on image data from the image sensors 314and 316. The processing apparatus 312 may include one or more processorshaving single or multiple processing cores. The processing apparatus 312may include memory, such as a random-access memory device (RAM), flashmemory, or another suitable type of storage device such as anon-transitory computer-readable memory. The memory of the processingapparatus 312 may include executable instructions and data that can beaccessed by one or more processors of the processing apparatus 312.

For example, the processing apparatus 312 may include one or moredynamic random access memory (DRAM) modules, such as double data ratesynchronous dynamic random-access memory (DDR SDRAM). In someimplementations, the processing apparatus 312 may include a digitalsignal processor (DSP). In some implementations, the processingapparatus 312 may include an application specific integrated circuit(ASIC). For example, the processing apparatus 312 may include a customimage signal processor.

The first image sensor 314 and the second image sensor 316 may beconfigured to detect light of a certain spectrum (e.g., the visiblespectrum or the infrared spectrum) and convey information constitutingan image as electrical signals (e.g., analog or digital signals). Forexample, the image sensors 314 and 316 may include CCDs or active pixelsensors in a CMOS. The image sensors 314 and 316 may detect lightincident through a respective lens (e.g., a fisheye lens). In someimplementations, the image sensors 314 and 316 include digital-to-analogconverters. In some implementations, the image sensors 314 and 316 areheld in a fixed orientation with respective fields of view that overlap.

The communications interface 318 may enable communications with apersonal computing device (e.g., a smartphone, a tablet, a laptopcomputer, or a desktop computer). For example, the communicationsinterface 318 may be used to receive commands controlling image captureand processing in the image capture device 310. For example, thecommunications interface 318 may be used to transfer image data to apersonal computing device. For example, the communications interface 318may include a wired interface, such as a high-definition multimediainterface (HDMI), a universal serial bus (USB) interface, or a FireWireinterface. For example, the communications interface 318 may include awireless interface, such as a Bluetooth interface, a ZigBee interface,and/or a Wi-Fi interface.

The user interface 320 may include an LCD display for presenting imagesand/or messages to a user. For example, the user interface 320 mayinclude a button or switch enabling a person to manually turn the imagecapture device 310 on and off. For example, the user interface 320 mayinclude a shutter button for snapping pictures.

The battery 322 may power the image capture device 310 and/or itsperipherals. For example, the battery 322 may be charged wirelessly orthrough a micro-USB interface.

The image capture system 300 may be used to implement some or all of thein-camera image processing and coding pipelines and techniques describedin this disclosure.

Referring to FIG. 3B, another image capture system 330 is shown. Theimage capture system 330 includes an image capture device 340 and apersonal computing device 360 that communicate via a communications link350. The image capture device 340 may, for example, be the image capturedevice 100 shown in FIGS. 1A-D. The personal computing device 360 may,for example, be the user interface device described with respect toFIGS. 1A-D.

The image capture device 340 includes an image sensor 342 that isconfigured to capture images. The image capture device 340 includes acommunications interface 344 configured to transfer images via thecommunication link 350 to the personal computing device 360.

The personal computing device 360 includes a processing apparatus 362that is configured to receive, using a communications interface 366,images from the image sensor 342. The processing apparatus 362 may beconfigured to perform image signal processing (e.g., filtering, tonemapping, stitching, and/or encoding) to generate output images based onimage data from the image sensor 342.

The image sensor 342 is configured to detect light of a certain spectrum(e.g., the visible spectrum or the infrared spectrum) and conveyinformation constituting an image as electrical signals (e.g., analog ordigital signals). For example, the image sensor 342 may include CCDs oractive pixel sensors in a CMOS. The image sensor 342 may detect lightincident through a respective lens (e.g., a fisheye lens). In someimplementations, the image sensor 342 includes digital-to-analogconverters. Image signals from the image sensor 342 may be passed toother components of the image capture device 340 via a bus 346.

The communications link 350 may be a wired communications link or awireless communications link. The communications interface 344 and thecommunications interface 366 may enable communications over thecommunications link 350. For example, the communications interface 344and the communications interface 366 may include an HDMI port or otherinterface, a USB port or other interface, a FireWire interface, aBluetooth interface, a ZigBee interface, and/or a Wi-Fi interface. Forexample, the communications interface 344 and the communicationsinterface 366 may be used to transfer image data from the image capturedevice 340 to the personal computing device 360 for image signalprocessing (e.g., filtering, tone mapping, stitching, and/or encoding)to generate output images based on image data from the image sensor 342.

The processing apparatus 362 may include one or more processors havingsingle or multiple processing cores. The processing apparatus 362 mayinclude memory, such as RAM, flash memory, or another suitable type ofstorage device such as a non-transitory computer-readable memory. Thememory of the processing apparatus 362 may include executableinstructions and data that can be accessed by one or more processors ofthe processing apparatus 362. For example, the processing apparatus 362may include one or more DRAM modules, such as DDR SDRAM.

In some implementations, the processing apparatus 362 may include a DSP.In some implementations, the processing apparatus 362 may include anintegrated circuit, for example, an ASIC. For example, the processingapparatus 362 may include a custom image signal processor. Theprocessing apparatus 362 may exchange data (e.g., image data) with othercomponents of the personal computing device 360 via a bus 368.

The personal computing device 360 may include a user interface 364. Forexample, the user interface 364 may include a touchscreen display forpresenting images and/or messages to a user and receiving commands froma user. For example, the user interface 364 may include a button orswitch enabling a person to manually turn the personal computing device360 on and off. In some implementations, commands (e.g., start recordingvideo, stop recording video, or capture photo) received via the userinterface 364 may be passed on to the image capture device 340 via thecommunications link 350.

The image capture system 330 may be used to implement some or all of thein-camera image processing and coding pipelines and techniques describedin this disclosure.

FIG. 4 is a functional block diagram of an example of an in-camera imageprocessing and coding pipeline portion 400 for night lapse video. Theimage processing and coding pipeline portion 400 may be included in animage capture device, such as the image capture device 100 shown inFIGS. 1A-D, the image capture device 200 shown in FIGS. 2A-C, the imagecapture system 300 shown in FIG. 3A, or the image capture system 330shown in FIG. 3B. The portion of the image processing and codingpipeline portion 400 shown in FIG. 4 may implement three-dimensionalnoise reduction including electronic image stabilization, which mayinclude image quality enhancements and power efficiency enhancements fornight lapse video.

The image processing and coding pipeline portion 400 may receive animage signal 402, such as from one or more image sensors (not shown),such as the image sensor 314 shown in FIG. 3A. The one or more imagesensors may be configured to support an integration time of 30 secondsor more. The image signal 402 may be received as input image data. Theimage signal 402 may be received in a defined format, such an unsigned14-bit Bayer format.

As shown in FIG. 4 , the image processing and coding pipeline portion400 includes a sensor input (SEN) unit 410 for receiving the imagesignal 402 as input image data. The sensor input unit 410 may processthe input image data and may output image data.

As shown in FIG. 4 , the image processing unit 420 receives the imagedata from the sensor input unit 410, a memory 425, or both. The imageprocessing unit 420 may convert the received image data to an unsigned10-bit YUV image data. The image processing unit 420 may store theunsigned 10-bit YUV image data in memory 425. The image processing unit420 may be configured to perform any number of passes. For example, insome implementations, the image processing unit 420 may be configured toperform two or more passes.

As shown in FIG. 4 , the image processing and coding pipeline portion400 includes a temporal noise reduction unit 430. In this example, thetemporal noise reduction unit 430 may be a three-dimensional noisereduction (3DNR) unit. The image processing unit 420 may output theimage data to the temporal noise reduction unit 430 and the temporalnoise reduction unit 430 may receive the image data from the imageprocessing unit 420. The image data received at the temporal noisereduction unit 430 may be unsigned 10-bit YUV image data. The receivedimage data may include one or more images. Each received image may havea corresponding reference image. The reference image may temporallyprecede the received image, and in some implementations, the referenceimage may be adjacent to and immediately precede the received image.

The temporal noise reduction unit 430 may determine a motion estimationbased on a comparison of a portion of the received image and a portionof the reference image. The received image and the reference image maybe divided into a number of blocks or portions, for example, each imagemay be divided into 16×16 blocks. The portion of the received image maycorrespond to the portion of the reference image such that the pixels ofthe portion of the received image are the same pixels of the portion ofthe reference image. The temporal noise reduction unit 430 may obtainmotion vectors based on a comparison of the pixels of the portion of thereceived image with the pixels in the corresponding portion of thereference image, a time reference relative to the reference frame, orboth, to determine the motion estimation. The motion estimation mayidentify pixels that are determined to be moving in the received imagerelative to the reference image. The temporal noise reduction unit 430may obtain a mask based on the motion estimation, for example, thepixels that are identified to be moving. The temporal noise reductionunit 430 may subtract the mask from the received image to obtain adenoised image. In some implementations, the temporal noise reductionunit 430 may be configured to perform spatial noise reduction.

As shown in FIG. 4 , the image processing and coding pipeline portion400 includes a video encoder (VE) 440. The video encoder 440 may be anytype of video encoder configured to encode video into any type of videoformat. For example, the video encoder 440 may be an advanced videocoding (AVC) encoder configured to encode video into an H.264 format ora high-efficiency video coding (HEVC) encoder configured to encode videointo an H.265 format. The video encoder 440 may be configured to encodethe denoised image at a bitrate from about 60 Mbps to about 120 Mbps. Inan example, the video encoder 440 may be configured to dynamicallyselect a bitrate prior to the image capture, during the image capture,or both. The video encoder 440 may be configured to dynamically selectthe bitrate based on a function of ISO, a function of integration time,a function of a scene classification, or any combination thereof.Example scene classifications may include a night sky, a sunrise, asunset, a cityscape, or the like.

FIG. 5 is a functional block diagram of another example of an in-cameraimage processing and coding pipeline portion 500 for night lapse video.The image processing and coding pipeline portion 500 may be included inan image capture device, such as the image capture device 100 shown inFIGS. 1A-D, the image capture device 200 shown in FIGS. 2A-C, the imagecapture system 300 shown in FIG. 3A, or the image capture system 330shown in FIG. 3B. The image processing and coding pipeline portion 500may be similar to the image processing and coding pipelines, or pipelineportions, shown in FIG. 4 , except as described herein or otherwiseclear from context. The portion of the image processing and codingpipeline portion 500 shown in FIG. 5 may implement three-dimensionalnoise reduction including electronic image stabilization, which mayinclude image quality enhancements and power efficiency enhancements fornight lapse video.

The image processing and coding pipeline portion 500 may receive animage signal 502, such as from one or more image sensors (not shown),such as the image sensor 314 shown in FIG. 3A. The one or more imagesensors may be configured to support an integration time of 30 secondsor more. The image signal 502 may be received as input image data. Theimage signal 502 may be received in a defined format, such an unsigned14-bit Bayer format.

As shown in FIG. 5 , the image processing and coding pipeline portion500 includes a sensor input unit 510 for receiving the image signal 502as input image data. The sensor input unit 510 may be similar to thesensor input unit 410 shown in FIG. 4 , except as described herein orotherwise clear from context. The sensor input unit 510 may process theinput image data and may output image data.

As shown in FIG. 5 , the image processing and coding pipeline portion500 includes a sensor read out unit 520. The sensor input unit 510 mayoutput the input image data to the sensor read out unit 520 and thesensor read out unit 520 may receive the input image data from thesensor input unit 510. The sensor read out unit 520 may output the imagedata as 15-bit Bayer format data. The image data output by the sensorread out unit 520 may be stored in memory 525, which may includecompression of the image data. The sensor read out unit 520 may beconfigured to perform any number of passes. For example, in someimplementations, the sensor read out unit 520 may be configured toperform two or more passes.

As shown in FIG. 5 , the image processing and coding pipeline portion500 includes a Bayer-to-Bayer unit 530. The sensor read out unit 520 mayoutput the image data to the Bayer-to-Bayer unit 530 and theBayer-to-Bayer unit 530 may receive the image data from the sensor readout unit 520, the memory 525, or both. The Bayer-to-Bayer unit 530 mayconvert the 15-bit Bayer format data to the 14-bit Bayer format. TheBayer-to-Bayer unit 530 may output the image data as 14-bit Bayer formatdata. The image data output by the Bayer-to-Bayer unit 530 may be storedin memory 525. The Bayer-to-Bayer unit 530 may include a noise reductionunit, for example a two-dimensional noise reduction unit. TheBayer-to-Bayer unit 530 may be configured to perform any number ofpasses. For example, in some implementations, the Bayer-to-Bayer unit530 may be configured to perform two or more passes.

As shown in FIG. 5 , the portion of the image processing and codingpipeline portion 500 includes a Bayer-to-RGB unit 540. TheBayer-to-Bayer unit 530 may output the image data to the Bayer-to-RGBunit 540 and the Bayer-to-RGB unit 540 may receive the image data fromthe Bayer-to-Bayer unit 530, the memory 525, or both. The Bayer-to-RGBunit 540 may convert the image data from Bayer format to an RGB format,such as a 14-bit RGB format. The image data output by the Bayer-to-RGBunit 540 may be stored in memory 525. The Bayer-to-RGB unit 540 mayinclude a noise reduction unit, for example a two-dimensional noisereduction unit. The Bayer-to-RGB unit 540 may be configured to performany number of passes. For example, in some implementations, theBayer-to-RGB unit 540 may be configured to perform two or more passes.

As shown in FIG. 5 , the image processing and coding pipeline portion500 includes an RGB-to-YUV unit 550. The Bayer-to-RGB unit 540 mayoutput the image data to the RGB-to-YUV unit 550 and the RGB-to-YUV unit550 may receive the image data from the Bayer-to-RGB unit 540, thememory 525, or both. The RGB-to-YUV unit 550 may convert the image datafrom RGB format to a YUV format, such as a 10-bit YUV format. The imagedata output by the RGB-to-YUV unit 550 may be stored in memory 525. TheRGB-to-YUV unit 550 may be configured to perform any number of passes.For example, in some implementations, the RGB-to-YUV unit 550 may beconfigured to perform two or more passes.

As shown in FIG. 5 , the image processing and coding pipeline portion500 includes a Chroma Noise Reduction unit 560. The RGB-to-YUV unit 550may output the image data to the Chroma Noise Reduction unit 560 and theChroma Noise Reduction unit 560 may receive the image data from theRGB-to-YUV unit 550, the memory 525, or both. The image data received atthe Chroma Noise Reduction unit 560 may be unsigned 10-bit YUV imagedata. The Chroma Noise Reduction unit 560 may output partially processedimage data as indicated at 565. The Chroma Noise Reduction unit 560 maybe configured to perform any number of passes. For example, in someimplementations, the Chroma Noise Reduction unit 560 may be configuredto perform two or more passes.

As shown in FIG. 5 , the image processing and coding pipeline portion500 includes a smart geometrical distortion engine (SGDE) 570. The SGDE570 is configured to correct distortions such as warping caused bysensor tilt, lens barrel distortion, or both. The Chroma Noise Reductionunit 560 may output the image data to the SGDE 570 and the SGDE 570 mayreceive the image data from the Chroma Noise Reduction unit 560. Theimage data received at the SGDE 570 may be unsigned 10-bit YUV imagedata.

As shown in FIG. 5 , the image processing and coding pipeline portion500 includes a temporal noise reduction unit 580. In this example, thetemporal noise reduction unit 580 may be a three-dimensional noisereduction unit. The SGDE 570 may output the image data to the temporalnoise reduction unit 580 and the temporal noise reduction unit 580 mayreceive the image data from the SGDE 570. The image data received at thetemporal noise reduction unit 580 may be unsigned 10-bit YUV image data.The received image data may include one or more images. Each receivedimage may have a corresponding reference image. The reference image maytemporally precede the received image, and in some implementations, thereference image may be adjacent to and immediately precede the receivedimage.

The temporal noise reduction unit 580 may determine a motion estimationbased on a comparison of a portion of the received image and a portionof the reference image. The received image and the reference image maybe divided into a number of blocks or portions, for example, each imagemay be divided into 16×16 blocks. The portion of the received image maycorrespond to the portion of the reference image such that the pixels ofthe portion of the received image are the same pixels of the portion ofthe reference image. The temporal noise reduction unit 580 may obtainmotion vectors based on a comparison of the pixels of the portion of thereceived image with the pixels in the corresponding portion of thereference image, a time reference relative to the reference frame, orboth, to determine the motion estimation. The motion estimation mayidentify pixels that are determined to be moving in the received imagerelative to the reference image. The temporal noise reduction unit 580may obtain a mask based on the motion estimation, for example, thepixels that are identified to be moving. The temporal noise reductionunit 580 may subtract the mask from the received image to obtain adenoised image. In some implementations, the temporal noise reductionunit 580 may be configured to perform spatial noise reduction.

As shown in FIG. 5 , the image processing and coding pipeline portion500 includes a video encoder 590. The video encoder 590 may be any typeof video encoder configured to encode video into any type of videoformat. For example, the video encoder 590 may be an AVC encoderconfigured to encode video into an H.264 format or an HEVC encoderconfigured to encode video into an H.265 format. The video encoder 590may be configured to encode the denoised image at a bitrate from about60 Mbps to about 120 Mbps. In an example, the video encoder 590 may beconfigured to dynamically select a bitrate prior to the image capture,during the image capture, or both. The video encoder 590 may beconfigured to dynamically select the bitrate based on a function of ISO,a function of integration time, a function of a scene classification, orany combination thereof. Example scene classifications may include anight sky, a sunrise, a sunset, a cityscape, or the like.

FIG. 6 is a functional block diagram of another example of an in-cameraimage processing and coding pipeline portion 600 for night lapse video.The image processing and coding pipeline portion 600 may be included inan image capture device, such as the image capture device 100 shown inFIGS. 1A-D, the image capture device 200 shown in FIGS. 2A-C, the imagecapture system 300 shown in FIG. 3A, or the image capture system 330shown in FIG. 3B. The image processing and coding pipeline portion 600may be similar to the image processing and coding pipelines, or pipelineportions, shown in FIG. 4 , except as described herein or otherwiseclear from context. The portion of the image processing and codingpipeline portion 600 shown in FIG. 6 may implement local tone mappingand three-dimensional noise reduction including electronic imagestabilization, which may include image quality enhancements and powerefficiency enhancements for night lapse video.

The image processing and coding pipeline portion 600 may receive animage signal 602, such as from one or more image sensors (not shown),such as the image sensor 314 shown in FIG. 3A. The one or more imagesensors may be configured to support an integration time of 30 secondsor more. The image signal 602 may be received as input image data. Theimage signal 602 may be received in a defined format, such an unsigned14-bit Bayer format.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes a sensor input unit 610 for receiving the image signal 602as input image data. The sensor input unit 610 may be similar to thesensor input unit 410 shown in FIG. 4 , except as described herein orotherwise clear from context. The sensor input unit 610 may process theinput image data and may output image data.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes a sensor read out unit 620. The sensor input unit 610 mayoutput the input image data to the sensor read out unit 620 and thesensor read out unit 620 may receive the input image data from thesensor input unit 610. The sensor read out unit 620 may output the imagedata as 15-bit Bayer format data. The image data output by the sensorread out unit 620 may be stored in memory 625, which may includecompression of the image data. The sensor read out unit 620 may beconfigured to perform any number of passes. For example, in someimplementations, the sensor read out unit 620 may be configured toperform two or more passes.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes a Bayer-to-Bayer unit 630. The sensor read out unit 620 mayoutput the image data to the Bayer-to-Bayer unit 630 and theBayer-to-Bayer unit 630 may receive the image data from the sensor readout unit 620, the memory 625, or both. The Bayer-to-Bayer unit 630 mayconvert the 15-bit Bayer format data to the 14-bit Bayer format. TheBayer-to-Bayer unit 630 may output the image data as 14-bit Bayer formatdata. The image data output by the Bayer-to-Bayer unit 630 may be storedin memory 625. The Bayer-to-Bayer unit 630 may include a noise reductionunit, for example a two-dimensional noise reduction unit. TheBayer-to-Bayer unit 630 may be configured to perform any number ofpasses. For example, in some implementations, the Bayer-to-Bayer unit630 may be configured to perform two or more passes.

As shown in FIG. 6 , the portion of the image processing and codingpipeline portion 600 includes a Bayer-to-RGB unit 640. TheBayer-to-Bayer unit 630 may output the image data to the Bayer-to-RGBunit 640 and the Bayer-to-RGB unit 640 may receive the image data fromthe Bayer-to-Bayer unit 630, the memory 625, or both. The Bayer-to-RGBunit 640 may convert the image data from Bayer format to an RGB format,such as a 14-bit RGB format. The image data output by the Bayer-to-RGBunit 640 may be stored in memory 625. The Bayer-to-RGB unit 640 mayinclude a noise reduction unit, for example a two-dimensional noisereduction unit. The Bayer-to-RGB unit 640 may be configured to performany number of passes. For example, in some implementations, theBayer-to-RGB unit 640 may be configured to perform two or more passes.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes a Local Tone Mapping (LTM) unit 645. The Bayer-to-RGB unit640 may output the image data to the Local Tone Mapping unit 645 and theLocal Tone Mapping unit 645 may receive the image data from theBayer-to-RGB unit 640, the memory 625, or both. The Local Tone Mappingunit 645 may perform local tone mapping for wide dynamic range (WDR)images. The Local Tone Mapping unit 645 may be configured to perform anynumber of passes. For example, in some implementations, the Local ToneMapping unit 645 may be configured to perform two or more passes.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes an RGB-to-YUV unit 650. The Local Tone Mapping unit 645 mayoutput the image data to the RGB-to-YUV unit 650 and the RGB-to-YUV unit650 may receive the image data from the Local Tone Mapping unit 645, thememory 625, or both. The RGB-to-YUV unit 650 may convert the image datafrom RGB format to a YUV format, such as a 10-bit YUV format. The imagedata output by the RGB-to-YUV unit 650 may be stored in memory 625. TheRGB-to-YUV unit 650 may be configured to perform any number of passes.For example, in some implementations, the RGB-to-YUV unit 650 may beconfigured to perform two or more passes.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes a Chroma Noise Reduction unit 660. The RGB-to-YUV unit 650may output the image data to the Chroma Noise Reduction unit 660 and theChroma Noise Reduction unit 660 may receive the image data from theRGB-to-YUV unit 650, the memory 625, or both. The image data received atthe Chroma Noise Reduction unit 660 may be unsigned 10-bit YUV imagedata. The Chroma Noise Reduction unit 660 may output partially processedimage data as indicated at 665. The Chroma Noise Reduction unit 660 maybe configured to perform any number of passes. For example, in someimplementations, the Chroma Noise Reduction unit 660 may be configuredto perform two or more passes.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes a smart geometrical distortion engine (SGDE) 670. The SGDE670 is configured to correct distortions such as warping caused bysensor tilt, lens barrel distortion, or both. The Chroma Noise Reductionunit 660 may output the image data to the SGDE 670 and the SGDE 670 mayreceive the image data from the Chroma Noise Reduction unit 660. Theimage data received at the SGDE 670 may be unsigned 10-bit YUV imagedata.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes a temporal noise reduction unit 680. In this example, thetemporal noise reduction unit 680 may be a 3DNR unit. The SGDE 670 mayoutput the image data to the temporal noise reduction unit 680 and thetemporal noise reduction unit 680 may receive the image data from theSGDE 670. The image data received at the temporal noise reduction unit680 may be unsigned 10-bit YUV image data. The received image data mayinclude one or more images. Each received image may have a correspondingreference image. The reference image may temporally precede the receivedimage, and in some implementations, the reference image may be adjacentto and immediately precede the received image.

The temporal noise reduction unit 680 may determine a motion estimationbased on a comparison of a portion of the received image and a portionof the reference image. The received image and the reference image maybe divided into a number of blocks or portions, for example, each imagemay be divided into 16×16 blocks. The portion of the received image maycorrespond to the portion of the reference image such that the pixels ofthe portion of the received image are the same pixels of the portion ofthe reference image. The temporal noise reduction unit 680 may obtainmotion vectors based on a comparison of the pixels of the portion of thereceived image with the pixels in the corresponding portion of thereference image, a time reference relative to the reference frame, orboth, to determine the motion estimation. The motion estimation mayidentify pixels that are determined to be moving in the received imagerelative to the reference image. The temporal noise reduction unit 680may obtain a mask based on the motion estimation, for example, thepixels that are identified to be moving. The temporal noise reductionunit 680 may subtract the mask from the received image to obtain adenoised image. In some implementations, the temporal noise reductionunit 680 may be configured to perform spatial noise reduction.

As shown in FIG. 6 , the image processing and coding pipeline portion600 includes a video encoder 690. The video encoder 690 may be any typeof video encoder configured to encode video into any type of videoformat. For example, the video encoder 690 may be an AVC encoderconfigured to encode video into an H.264 format or an HEVC encoderconfigured to encode video into an H.265 format. The video encoder 690may be configured to encode the denoised image at a bitrate from about60 Mbps to about 120 Mbps. In an example, the video encoder 690 may beconfigured to dynamically select a bitrate prior to the image capture,during the image capture, or both. The video encoder 690 may beconfigured to dynamically select the bitrate based on a function of ISO,a function of integration time, a function of a scene classification, orany combination thereof. Example scene classifications may include anight sky, a sunrise, a sunset, a cityscape, or the like.

FIG. 7 is a functional block diagram of an example of an in-camera videoprocessing and coding pipeline with real-time video preview. The imageprocessing and coding pipeline portion 700 may be included in an imagecapture device, such as the image capture device 100 shown in FIGS.1A-D, the image capture device 200 shown in FIGS. 2A-C, the imagecapture system 300 shown in FIG. 3A, or the image capture system 330shown in FIG. 3B. The image processing and coding pipeline portion 700includes an image processing portion 701 and a video preview portion703. The image processing portion 701 of the image processing and codingpipeline portion 700 may be similar to the image processing and codingpipelines, or pipeline portions, shown in FIG. 4 , except as describedherein or otherwise clear from context. The portion of the imageprocessing and coding pipeline portion 700 shown in FIG. 7 may implementlocal tone mapping and three-dimensional noise reduction includingelectronic image stabilization, which may include image qualityenhancements and power efficiency enhancements for night lapse video.

The image processing and coding pipeline portion 700 may receive animage signal 704, such as from one or more image sensors (not shown),such as the image sensor 314 shown in FIG. 3A. The one or more imagesensors may be configured to support an integration time of 30 secondsor more. The image signal 704 may be received as input image data. Theinput image signal 704 may be received in a defined format, such anunsigned 14-bit Bayer format.

As shown in FIG. 7 , the image processing portion 701 includes a sensorinput unit 710 for receiving the image signal 704 as input image data.The sensor input unit 710 may be similar to the sensor input unit 410shown in FIG. 4 , except as described herein or otherwise clear fromcontext. The sensor input unit 710 may process the input image data andmay output image data.

As shown in FIG. 7 , the image processing portion 701 includes a sensorread out unit 720. The sensor input unit 710 may output the input imagedata to the sensor read out unit 720 and the sensor read out unit 720may receive the input image data from the sensor input unit 710. Thesensor read out unit 720 may output the image data as 15-bit Bayerformat data. The image data output by the sensor read out unit 720 maybe stored in memory 725, which may include compression of the imagedata. The sensor read out unit 720 may be configured to perform anynumber of passes. For example, in some implementations, the sensor readout unit 720 may be configured to perform two or more passes. The imagedata may be used to compute statistics 710 a, for example auto-exposure(AE) statistics, auto-white balance (AWB) statistics, or both.

As shown in FIG. 7 , the image processing portion 701 includes aBayer-to-Bayer unit 730. The sensor read out unit 720 may output theimage data to the Bayer-to-Bayer unit 730 and the Bayer-to-Bayer unit730 may receive the image data from the sensor read out unit 720, thememory 725, or both. The Bayer-to-Bayer unit 730 may convert the 15-bitBayer format data to the 14-bit Bayer format. The Bayer-to-Bayer unit730 may output the image data as 14-bit Bayer format data. The imagedata output by the Bayer-to-Bayer unit 730 may be stored in memory 725.The Bayer-to-Bayer unit 730 may include a noise reduction unit, forexample a two-dimensional noise reduction unit. The Bayer-to-Bayer unit730 may be configured to perform any number of passes. For example, insome implementations, the Bayer-to-Bayer unit 730 may be configured toperform two or more passes.

As shown in FIG. 7 , the portion of the image processing portion 701includes a Bayer-to-RGB unit 740. The Bayer-to-Bayer unit 730 may outputthe image data to the Bayer-to-RGB unit 740 and the Bayer-to-RGB unit740 may receive the image data from the Bayer-to-Bayer unit 730, thememory 725, or both. The Bayer-to-RGB unit 740 may convert the imagedata from Bayer format to an RGB format, such as a 14-bit RGB format.The image data output by the Bayer-to-RGB unit 740 may be stored inmemory 725. The Bayer-to-RGB unit 740 may include a noise reductionunit, for example a two-dimensional noise reduction unit. TheBayer-to-RGB unit 740 may be configured to perform any number of passes.For example, in some implementations, the Bayer-to-RGB unit 740 may beconfigured to perform two or more passes.

As shown in FIG. 7 , the image processing portion 701 includes a LocalTone Mapping unit 745. The Local Tone Mapping unit 745 is shown in adashed line to indicate that some implementations may not include theLocal Tone Mapping unit 745. The Bayer-to-RGB unit 740 may output theimage data to the Local Tone Mapping unit 745 and the Local Tone Mappingunit 745 may receive the image data from the Bayer-to-RGB unit 740, thememory 725, or both. The Local Tone Mapping unit 745 may perform localtone mapping for wide dynamic range (WDR) images. The Local Tone Mappingunit 745 may be configured to perform any number of passes. For example,in some implementations, the Local Tone Mapping unit 745 may beconfigured to perform two or more passes.

As shown in FIG. 7 , the image processing portion 701 includes anRGB-to-YUV unit 750. The Local Tone Mapping unit 745 may output theimage data to the RGB-to-YUV unit 750 and the RGB-to-YUV unit 750 mayreceive the image data from the Local Tone Mapping unit 745, the memory725, or both. The RGB-to-YUV unit 750 may convert the image data fromRGB format to a YUV format, such as a 10-bit YUV format. The image dataoutput by the RGB-to-YUV unit 750 may be stored in memory 725. TheRGB-to-YUV unit 750 may be configured to perform any number of passes.For example, in some implementations, the RGB-to-YUV unit 750 may beconfigured to perform two or more passes.

As shown in FIG. 7 , the image processing portion 701 includes a ChromaNoise Reduction unit 760. The RGB-to-YUV unit 750 may output the imagedata to the Chroma Noise Reduction unit 760 and the Chroma NoiseReduction unit 760 may receive the image data from the RGB-to-YUV unit750, the memory 725, or both. The image data received at the ChromaNoise Reduction unit 760 may be unsigned 10-bit YUV image data. TheChroma Noise Reduction unit 760 may output partially processed imagedata as indicated at 765. The Chroma Noise Reduction unit 760 may beconfigured to perform any number of passes. For example, in someimplementations, the Chroma Noise Reduction unit 760 may be configuredto perform two or more passes.

As shown in FIG. 7 , the image processing portion 701 includes a smartgeometrical distortion engine (SGDE) 770. The SGDE 770 is configured tocorrect distortions such as warping caused by sensor tilt, lens barreldistortion, or both. The Chroma Noise Reduction unit 760 may output theimage data to the SGDE 770 and the SGDE 770 may receive the image datafrom the Chroma Noise Reduction unit 760. The image data received at theSGDE 770 may be unsigned 10-bit YUV image data.

As shown in FIG. 7 , the image processing portion 701 includes atemporal noise reduction unit 780. In this example, the temporal noisereduction unit 780 may be a 3DNR unit. The SGDE 770 may output the imagedata to the temporal noise reduction unit 780 and the temporal noisereduction unit 780 may receive the image data from the SGDE 770. Theimage data received at the temporal noise reduction unit 780 may beunsigned 10-bit YUV image data. The received image data may include oneor more images. Each received image may have a corresponding referenceimage. The reference image may temporally precede the received image,and in some implementations, the reference image may be adjacent to andimmediately precede the received image.

The temporal noise reduction unit 780 may determine a motion estimationbased on a comparison of a portion of the received image and a portionof the reference image. The received image and the reference image maybe divided into a number of blocks or portions, for example, each imagemay be divided into 16×16 blocks. The portion of the received image maycorrespond to the portion of the reference image such that the pixels ofthe portion of the received image are the same pixels of the portion ofthe reference image. The temporal noise reduction unit 780 may obtainmotion vectors based on a comparison of the pixels of the portion of thereceived image with the pixels in the corresponding portion of thereference image, a time reference relative to the reference frame, orboth, to determine the motion estimation. The motion estimation mayidentify pixels that are determined to be moving in the received imagerelative to the reference image. The temporal noise reduction unit 780may obtain a mask based on the motion estimation, for example, thepixels that are identified to be moving. The temporal noise reductionunit 780 may subtract the mask from the received image to obtain adenoised image. In some implementations, the temporal noise reductionunit 780 may be configured to perform spatial noise reduction.

As shown in FIG. 7 , the image processing portion 701 includes a videoencoder 790. The video encoder 790 may be any type of video encoderconfigured to encode video into any type of video format. For example,the video encoder 790 may be an AVC encoder configured to encode videointo an H.264 format or an HEVC encoder configured to encode video intoan H.265 format. The video encoder 790 may be configured to encode thedenoised image at a bitrate from about 60 Mbps to about 120 Mbps. In anexample, the video encoder 790 may be configured to dynamically select abitrate prior to the image capture, during the image capture, or both.The video encoder 790 may be configured to dynamically select thebitrate based on a function of ISO, a function of integration time, afunction of a scene classification, or any combination thereof. Examplescene classifications may include a night sky, a sunrise, a sunset, acityscape, or the like.

As shown in FIG. 7 , the video preview portion 703 includes an imageprocessing unit 791 that performs the functions of the sensor readoutunit 720, the Bayer-to-Bayer unit 730, the Bayer-to-RGB unit 740, theRGB-to-YUV unit 750, and the chroma noise reduction unit 760. The imageprocessing unit 791 may be configured to perform any number of passes.For example, in some implementations, the image processing unit 791 maybe configured to perform two or more passes. The output of the imageprocessing unit 791 may be used to compute statistics 791 a, for exampleglobal tone mapping (GTM) statistics. The output of the image processingunit 791 may be used to perform face detection (FD) 791 b. The output ofthe image processing unit 791 is sent to an integrated performanceprimitive (IPP) unit 792. The IPP unit 792 receives the output from theimage processing unit 791 and performs a warping correction, imagestabilization, or both. The output of the IPP unit 792 may be sent to amotion estimation (ME) unit 793. The motion estimation unit 793 mayperform motion estimation as described above. The output from the motionestimation unit 793 may be used to calculate one or more local motionvectors (LMV)s 793 a. The output of the IPP unit 792 may be sent to avideo encoder 794. The video encoder 794 may receive and process theoutput of the IPP unit 792 using AVC or HEVC to output a preview videofor display on the image capture device in real-time during a nightlapse video capture.

The output of the IPP unit 792 may be sent to an IPP unit 795. The IPPunit 795 receives the output of the IPP unit 792 and rasterizes androtates the image data based on an orientation of the image capturedevice. The output of the IPP unit 795 may be sent to the display 796 ofthe image capture device or a high-definition multi-media interface(HDMI) for display. The output of the IPP unit 795 may be used todetermine a scene classification 795 a.

FIG. 8 is a flow diagram of an in-camera video processing method 800 fornight lapse video. The method may include capturing 810 image data,denoising 820 an image, encoding 830 the denoised image, and outputting840 a video file.

Referring to FIG. 8 , capturing 810 image data may include capturing animage. Capturing 810 image data may include capturing a reference imagethat is temporally precedent to the image. In an example, there may be aone-frame delay between the image and the reference image. The referenceimage may be a frame that is immediately preceding the image. In someexamples, the reference image may precede the image by two or moreframes. The image and the reference image may be processed using animage processing pipeline.

Processing the image, the reference image, or both, using an imageprocessing pipeline may include converting a 15-bit Bayer image to a14-bit Bayer image. Processing the image, the reference image, or both,using an image processing pipeline may include converting the 14-bitBayer image to a 14-bit RGB image. Processing the image, the referenceimage, or both, using an image processing pipeline may includeconverting the 14-bit RGB image to a 10-bit YUV image. Processing theimage, the reference image, or both, using an image processing pipelinemay include performing chroma denoising of the 10-bit YUV image.Processing the image, the reference image, or both, using an imageprocessing pipeline may include performing a warp correction of the10-bit YUV image.

Denoising 820 the image, the reference image, or both, may includedetermining a motion estimation. The motion estimation may be based on acomparison of a portion of the image and a corresponding portion of thereference image. Denoising 820 the image, the reference image, or both,may include obtaining a mask based on the motion estimation. Denoising820 the image, the reference image, or both, may include subtracting themask from the image to obtain a denoised image.

Encoding 830 the denoised image may include encoding the denoised imagein a video file in an AVC format. Encoding 830 the denoised image mayinclude encoding the denoised image in a video file in an HEVC format.The denoised image may be encoded at a bitrate from about 60 Mbps toabout 120 Mbps. The bitrate may be dynamically selected prior to theimage capture or during the image capture.

Outputting 840 the video file may include storing the video file in amemory. Outputting 840 the video file may include displaying the videoon a display of the image capture device. Outputting 840 the video filemay include transmitting the video file for display on a remote device.

While the disclosure has been described in connection with certainembodiments, it is to be understood that the disclosure is not to belimited to the disclosed embodiments but, on the contrary, is intendedto cover various modifications and equivalent arrangements includedwithin the scope of the appended claims, which scope is to be accordedthe broadest interpretation so as to encompass all such modificationsand equivalent structures as is permitted under the law.

What is claimed is:
 1. A method comprising: obtaining an image and areference image; processing the image and the reference image using animage processing pipeline, wherein processing the image includes:converting the image to a YUV image; denoising the YUV image to obtain adenoised image; encoding the denoised image in a video format; andstoring a video file in the video format.
 2. The method of claim 1,wherein the image is a Bayer image, and wherein processing the imageincludes: performing chroma denoising of the YUV image; and performing awarp correction of the YUV image.
 3. The method of claim 2, wherein thereference image is a Bayer reference image, and wherein processing thereference image comprises: converting the Bayer reference image to a YUVreference image.
 4. The method of claim 3, wherein processing thereference image includes: performing chroma denoising of the referenceimage; and performing a warp correction of the reference image.
 5. Themethod of claim 4, wherein denoising the YUV image comprises:determining a motion estimation based on a comparison of a portion ofthe image and a portion of the reference image; obtaining a mask basedon the motion estimation; and subtracting the mask from the image toobtain a denoised image.
 6. The method of claim 1, wherein the videofile is encoded in an advanced video coding (AVC) format or ahigh-efficiency video coding (HEVC) format.
 7. The method of claim 1,wherein processing the image includes performing local tone mapping ofthe YUV image to obtain a wide dynamic range (WDR) image.
 8. The methodof claim 7, wherein obtaining the image includes obtaining the image atmultiple exposures.
 9. An image capture device comprising: an imagesensor configured to obtain an image and a reference image; a processorconfigured to process the image and the reference image using an imageprocessing pipeline, the processor configured to convert the referenceimage to a YUV reference image; a denoiser circuit configured to denoisethe image based on a motion estimation based on a comparison of aportion of the image and a portion of the reference image; and anencoder circuit configured to: encode the denoised image in a videoformat; and store a video file in the video format.
 10. The imagecapture device of claim 9, the processor further configured to: convertthe image to a YUV image; perform chroma denoising of the YUV image; andperform a warp correction of the YUV image.
 11. The image capture deviceof claim 10, the processor further configured to: perform chromadenoising of the YUV reference image; and perform a warp correction ofthe YUV reference image.
 12. The image capture device of claim 11,wherein the denoiser circuit is further configured to: obtain a maskbased on the motion estimation; and subtract the mask from the image toobtain a denoised image.
 13. The image capture device of claim 9,wherein the video file is encoded in an advanced video coding (AVC)format or a high-efficiency video coding (HEVC) format.
 14. The imagecapture device of claim 9, wherein the processor is further configuredto: perform local tone mapping of the YUV reference image to obtain awide dynamic range (WDR) reference image.
 15. The image capture deviceof claim 14, wherein the image sensor is configured to obtain the imageat multiple exposures.
 16. An image capture device comprising: an imagesensor configured to obtain an image and a reference image; a processorconfigured to process the image and the reference image using an imageprocessing pipeline, the processor configured to: obtain a YUV image;and obtain a YUV reference image; a denoiser circuit configured todenoise the YUV image to obtain a denoised image; and an encoder circuitconfigured to: encode the denoised image in a video format; and store avideo file in the video format.
 17. The image capture device of claim16, wherein the encoder circuit is an advanced video coding (AVC)encoder circuit or a high-efficiency video coding (HEVC) encodercircuit.
 18. The image capture device of claim 16, wherein the encodercircuit is further configured to select a bitrate prior to an imagecapture or during the image capture.
 19. The image capture device ofclaim 18, wherein the encoder circuit is further configured to selectthe bitrate based on a function of ISO or integration time.
 20. Theimage capture device of claim 18, wherein the encoder circuit is furtherconfigured to select the bitrate based on a function of a sceneclassification.